Repository navigation
fix(adapters): fix exception mapping, transactions, and settings consistency across adapters - #875
Merged
Merged
Conversation
Delete bool_to_int, resolve_column_names, detect_json_columns, and detect_json_columns_from_description from the aiomysql, asyncmy, mysqlconnector, and pymysql cores; the coercion maps register int directly. Drop the matching source-equivalence cases and the mysqlconnector create_mapped_exception exemption, since all four mappers are identical.
…ng transaction errors The MySQL exception mapper returned True for duplicate-key-name and cannot-drop errors, so the driver handlers suppressed the exception without a result and statement execution failed with an AssertionError. Map both codes to OperationalError, make every handler record the mapped exception, and make begin/commit/rollback always raise the mapped error with its cause.
…ation The Litestar session migration derives the store class from the config class name, so SqliteConfig, AdbcConfig, and DuckDBConfig failed to resolve SQLiteStore, ADBCStore, and DuckdbStore. Rename those stores, the DuckDB ADK classes, and the single-mode Sybase and SQL Anywhere event stores to match their configs, drop the old event store aliases, and add a guard test covering every adapter with a Litestar store.
… rows The operation profile marked any DML with a returning clause as row-returning, including T-SQL OUTPUT ... INTO, which writes into a table variable. mssql_python now fetches rows whenever the cursor exposes a result set, so EXEC and DELETE ... OUTPUT results are not dropped.
…extension configs FlaskConfig and StarletteConfig (and so FastAPIConfig) omitted enable_correlation_middleware, correlation_header, correlation_headers, and auto_trace_headers even though both extensions read them. Declare them and add a contract test that every key an extension reads is declared on its config type.
…ramework extension Flask, Starlette, FastAPI, and Sanic defaulted enable_correlation_middleware to False while Litestar defaulted to True. All five now default to True; set the key to False to opt out.
…t null and check Update SQLite and AioSQLite extended constraint error codes so NOT NULL uses code 1299 and CHECK uses code 275. This prevents trigger aborts from being falsely classified as not-null violations.
…ion, and error mapping Prevent leaking dirty connections in connection pool release by retiring connections on rollback errors. Delegate cached query execution to cursor method to preserve ambient exceptions and format wire SQL, and map programming/interface errors and column names containing substring keywords accurately.
…ddings and parameterize event limit Use WeakSet to track configured connections across ADK and Litestar stores to avoid ABA pragma skipping after address reuse. Implement binary float32 packing and similarity search for memory embeddings, and parameterize LIMIT in ADK get_events.
…adata caching Hoist connection parameter filtering helpers and excluded keys to module level, eliminating per-call closures. Hoist cursor context in batch Arrow loading, cache version detection at class level in data dictionary, and memoize generated column DDL expressions per table.
…r mappings Delegate script execution to base class with rowid cache invalidation, remove redundant pass-through load_from_storage, replace truthy or in pool creation with explicit is None checks, remove dead Python 3.11 version check, and add missing error mappings for auth, range, toobig, and notadb.
…ed memory cache, and error mappings Roll back uncommitted transactions on exit in get_connection unless explicitly committed, track thread-local reentrancy depth, default memory database to cache=shared, delegate script execution to base driver with rowid cache invalidation, and map substring errors only when error_code is None.
…ze events limit Implement binary float32 packing and vector similarity ranking for SQLite ADK memory store, parameterize LIMIT in get_events, and retain connection ID tracking for non-weakref-capable sqlite3.Connection instances.
…raise SQLFileNotFoundError Memoize generated column expression parsing per table in SQLite and AioSQLite data dictionaries. Coalesce schema_name to 'main' in SQLite index and column introspection queries, and raise SQLFileNotFoundError when data dictionary queries are missing.
…int handling Hoist connection parameter filtering helpers, epoch constants, and schema lists to module scope. Normalize cached query parameters, hoist cursor context in Arrow loading, and implement direct savepoint create/release without script splitting or rowid cache invalidation.
…ion lifecycle Track active connection checkouts, prune dead thread connections on size inspection, default transaction mode to IMMEDIATE with ImproperConfigurationError validation, delegate commit and rollback to end_transaction, remove redundant load_from_storage, and remove unused logger.
…ache hit fast path Ensure LOAD DATA statements escape percent characters in identifiers for PyMySQL pyformat interpolation. Check literal percent escaping in cached queries so clean queries delegate directly to the driver fast path without redundant statement allocations.
…L upsert syntax Batch ADK memory entries with cursor.executemany rather than sequential execute calls in a loop. Drop redundant DROP INDEX statement from session table teardown and use VALUES(col) in Litestar session upsert statements for cross-version MySQL and MariaDB compatibility.
…te generator closures in compiled helpers Add O(1) fast path to escape_literal_percent that returns the SQL string unchanged when parameter and percent counts match. Eliminate generator expressions from localized core helpers to avoid heap closures under MyPyC, and offload LOCAL INFILE tempfile creation and payload writes to asyncio.to_thread.
…te script execution, and clean up connection handling Pass statement.statement_config to statement compilation and parameter validation across execution methods. Delegate dispatch_execute_script to the base driver implementation to emit statement observability events, close standalone connections on creation failures, catch AiomysqlPymysqlError consistently in transaction methods, and remove the redundant load_from_storage override.
Delegate _execute_cache_hit to super()._execute_cache_hit when cached SQL queries do not require literal percent escaping. This eliminates unnecessary statement and result allocations and restores direct cursor execution via collect_rows and resolve_rowcount.
… use cross-version upsert syntax Batch ADK memory entries with cursor.executemany rather than sequential execute calls in a loop. Escape %, _, and \\ wildcards in memory search with an explicit ESCAPE clause, use LONGTEXT for content_text in the memory table DDL, remove redundant DROP INDEX from Litestar session teardown, and use VALUES(col) in Litestar session upsert statements for cross-version MySQL and MariaDB compatibility.
… add object_name filter to privileges Combine VERSION(), @@sql_mode, and @@sql_quote_show_create scalar queries into a single SELECT statement and cache the resulting DDL context per driver session to minimize round-trips before SHOW CREATE. Add object_name filtering support to the privileges domain query in MySQL data dictionaries.
…d eliminate redundant dict copy Offload LOCAL INFILE temporary file creation and payload writing to asyncio.to_thread in _load_from_arrow_via_local_infile, ensuring asynchronous cleanup with shutil.rmtree on both success and failure. Eliminate redundant dict copy of driver_features in AsyncmyConfig.__init__.
…mports, and streamline execution hooks Add explicit MySQL errno mappings for 1054, 1146, 1040, 1290, and 1836 to handle asyncmy errors without SQLSTATE, remove Postgres 235xx codes, and close connection on SSCursor close error. Guard top-level asyncmy imports, relocate AsyncmySessionContext to config.py, set native Arrow/Parquet capabilities accurately to False, delegate script execution to base driver, and close standalone connections on creation failure.
…n causes in load_from_arrow, and map connection errors Guard top-level mysql-connector imports using import_optional to prevent ModuleNotFoundError when optional dependencies are missing. Delegate stream fetching and load_from_arrow statement executions through _run_with_exception_handler and _check_pending_exception to maintain traceback chaining and exception context. Map PoolError to ConnectionTimeoutError and disconnected InterfaceError / errno 2055 to DatabaseConnectionError.
…andle missing tables, and align store DDL Move conn.commit() inside the try block before rollback in sync append_event_and_update_state. Handle missing-table errors gracefully in sync and async ADK memory stores, remove redundant DROP INDEX from Litestar session teardown, use VALUES(col) in Litestar session upsert statements for cross-version MySQL and MariaDB compatibility, and align sync create_table lifecycle by removing redundant commit after execute_script.
…d shared metadata setup Verify that data_dictionary properties on sync and async mysqlconnector drivers cache their instances. Use the consolidated scalar DDL metadata query and object_name filtering from the shared MySQL data dictionary.
…lizers, and compile column resolution Precompute cursor_options in driver __init__ to avoid dictionary allocations on with_cursor calls. Cache _deserializer in stream source instances on initialization to eliminate per-chunk getattr lookups, and move _resolve_column_types and type code mappings into compiled core.py.
…, force unprepared DDL cursors, and clean config inheritance Map MySQL errnos 1146 and 1054 to OperationalError instead of SQLParsingError, remove dead escape_literal_percent exports from core, and delegate script execution to the base driver. Force an unprepared cursor for TRUNCATE and LOAD DATA LOCAL INFILE in load_from_arrow to prevent errno 1295 when prepared statements are configured. Clean up connection context slots and remove redundant provide_connection and provide_session overrides in MysqlConnectorAsyncConfig.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adapter-wide correctness and consistency fixes from a review of all 22 database adapters. This change encompasses the cross-adapter contract fixes, web-framework and driver settings alignment, SQLite, MySQL, SQL Server/Sybase/SQL Anywhere/Db2, Oracle, PostgreSQL/CockroachDB family fixes, columnar adapters (ADBC, Arrow ODBC, BigQuery, DuckDB, Spanner), and a repository-wide adapter import contract.
The changes fix crashes and silent failures in exception mapping and transaction handling, align store and settings naming so framework integrations resolve every adapter, and tighten hot paths in the compiled adapter modules without shrinking the mypyc surface.
Key Changes
Core and extensions
SQLProcessorno longer treats T-SQLOUTPUT ... INTOas returning client rows.FlaskConfigandStarletteConfig(and soFastAPIConfig) declareenable_correlation_middleware,correlation_header,correlation_headers, andauto_trace_headers, which the extensions already read; a contract test checks that every key an extension reads is declared.enable_correlation_middlewarenow defaults toTruein the Flask, Starlette, FastAPI, and Sanic extensions, matching Litestar.Settings
pgbouncerandpgbouncer_modedriver_features aliases (asyncpg, psycopg, psqlpy) and the asyncpgconnection_config["pgbouncer"]key;pooler_mode="transaction"is the single switch.use_pooltoenable_pooling(mssql_python, pymssql) and replaced psycopgnull_poolwithenable_pooling=False.SybaseADKConfig,SqlanywhereADKConfig,Db2LitestarConfig, andDb2EventsConfigTypedDicts.arrow_odbcenable_driver_poolingtoenable_poolingand BigQueryuse_query_and_waittoenable_query_and_wait.strict_type_coercionandarrow_extension_typesdriver feature aliases; useenable_strict_type_coercionandenable_arrow_extension_types.events_backendonArrowOdbcDriverFeaturesanddefault_query_job_configonBigQueryDriverFeatures.Adapter import contract
sqlspec.adapters.<name>imports that adapter's own driver eagerly in_typing.py; a missing driver raises the driver's ownModuleNotFoundErrorinstead of failing later on placeholder values. Every driver stays optional and any adapter works without the others installed.pyarrow,numpy,pgvector,rich,google-cloud-bigquery-storage) load on first use; importing the oracledb, duckdb, psycopg, and CockroachDB psycopg adapters no longer pulls them in. The centralsqlspec.typingshim gainsArrowException.re.errorwhen asyncmy is missing.tests/unit/test_import_time.pyenforces the contract for every registered adapter, anddocs/contributing/creating-adapters.mddocuments it for new adapters.sqlanydbjoins theextrasdependency group.Store naming
SQLiteStoretoSqliteStore,ADBCStoretoAdbcStore,DuckdbStoretoDuckDBStore, the DuckDB ADK classes toDuckDBADKStore,DuckDBADKMemoryStore,DuckDBADKConfig, andDuckDBADKFTSOptions, and the single-mode Sybase and SQL Anywhere event stores toSybaseEventQueueStoreandSqlanywhereEventQueueStore. The Litestar session-table migration derives the store name from the config name, so it now resolves every adapter; a guard test covers all adapters with a Litestar store.MySQL family (
aiomysql,asyncmy,mysqlconnector,pymysql)OperationalErrorinstead of being suppressed, which previously crashed statement execution with anAssertionError;begin,commit, androllbackalways raise the mapped error with its cause.mysql_common.pyinto each adapter'score.py.aiomysqlescapes%inLOAD DATAstatements; cached statement execution takes the cursor fast path again; literal-percent escaping has a count fast path; generator closures removed from compiled helpers.DROPDDL and use a MySQL/MariaDB-compatible upsert; ADK memory inserts are batched withexecutemany; LIKE wildcards are escaped.asyncmymaps additional errnos, guards its optional driver import, offloadsLOCAL INFILEpayload writes to a worker thread, and delegates script execution to the base driver.SQLite family (
sqlite,aiosqlite)sqlitetracks connection re-entrancy so nested scopes no longer end the outer transaction (commit-on-exit is unchanged), defaults:memory:to a shared cache, rejects invalid transaction modes withImproperConfigurationError, and only applies substring error matching when no error code is available.aiosqliteretires connections whose rollback fails instead of returning them to the pool, isolates ambient exceptions on cache hits, and mapsProgrammingErrorandInterfaceError.LIMIT; generated-column metadata is memoized and schema names default tomain.SQL Server, Sybase, SQL Anywhere, Db2
TransactionErrorfor SQL Anywhere) with their cause.mssql_pythondetects implicit transactions and rolls back on session exit, maps ODBC SQLSTATE codes, and fetchesOUTPUT ... INTOresults only when the cursor has a description;pymssqlmaps FreeTDS error codes and keeps an identical mapper.MERGEstatements useHOLDLOCK; ADK memory inserts are batched; unconfigured vector search is rejected; SQL Server data dictionaries implement the missing metadata methods.USER_NAME(); SQL Anywhere reportssupports_transactional_ddl=Falseand bounds its connection pool.Oracle (
oracledb)PostgreSQL and CockroachDB (
asyncpg,psycopg,psqlpy,cockroach_asyncpg,cockroach_psycopg)configurecallback runs after internal setup.auth_token_providerat pool creation so tokens rotate.Columnar (
adbc,arrow_odbc,bigquery,duckdb,spanner)ArrowException, begins transactions for atomic writes on SQL Server, and reports metadata capabilities.RetryErrorandGoogleAPIError, handles empty iterable schemas, preserves exception causes, and declaresjson_deserializer.on_connection_create. Native async Spanner support is unchanged.